Gan-based data synthesis for semi-supervised learning of a radar sensor

ABSTRACT

Examples disclosed herein relate to a method for semi-supervised training of a radar system. The method includes training a first radar network of the radar system with a first set of radar object detection labels corresponding to a first set of radar data, training a generative adversarial network (GAN) with the trained first radar network, synthesizing a training data set for a second radar network of the radar system with the trained GAN, training a second radar network with the synthesized training data set, and generating a second set of radar object detection labels based on the training of the second radar network.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application No.62/968,826, filed on Jan. 31, 2020, which is incorporated by referencein its entirety.

BACKGROUND

Autonomous driving is quickly moving from the realm of science fictionto becoming an achievable reality. Already in the market areAdvanced-Driver Assistance Systems (“ADAS”) that can automate, adapt,and enhance vehicles for safety and better driving. The next step willbe vehicles that increasingly assume control of driving functions, suchas steering, accelerating, braking, and monitoring the surroundingenvironment and adjusting driving conditions to, for example, avoidtraffic, crossing pedestrians, animals, and so on, by changing lanes ordecreasing speed when needed. The requirements for object and imagedetection are critical to enable the aforementioned enhancements,particularly to control and perform driving functions within a shortenough response time required to capture, process and turn the data intoaction. All these enhancements are to be achieved in autonomous drivingwhile ensuring accuracy, consistency and cost optimization for deployingin the vehicles.

An aspect of making this work is the ability to detect, identify, andclassify objects in the surrounding environment at the same or possiblyat an even better level than humans. Humans are adept at recognizing andperceiving the world around them with an extremely complex human visualsystem that essentially has two main functional parts: the eye and thebrain. In autonomous driving technologies, the eye may include acombination of multiple sensors, such as camera, radar, and lidar, whilethe brain may involve multiple artificial intelligence, machine learningand deep learning systems. The goal is to have full understanding of adynamic, fast-moving environment in real time and human-likeintelligence to act in response to changes in the environment.Therefore, there is a need for vehicular systems that provide improveddynamic, responsive, and intelligent functionality for autonomousdriving.

BRIEF DESCRIPTION OF THE DRAWINGS

The present application may be more fully appreciated in connection withthe following detailed description taken in conjunction with theaccompanying drawings, which are not drawn to scale and in which likereference characters refer to like parts throughout, and wherein:

FIG. 1 illustrates an example environment in which a beam steering radarin an autonomous vehicle is used to detect and identify objects,according to various implementations of the subject technology;

FIG. 2 illustrates an example network environment in which a radarsystem may be implemented in accordance with one or more implementationsof the subject technology;

FIG. 3 is a schematic diagram of a beam steering radar system as in FIG.2 and in accordance with one or more implementations of the subjecttechnology;

FIG. 4 is a schematic diagram of a multi-sensor fusion platform inaccordance with various examples;

FIG. 5 illustrates an example range-doppler map captured by a beamsteering radar system as in FIG. 3 for an outdoor scene and inaccordance with one or more implementations of the subject technology;

FIG. 6 is a schematic diagram illustrating various sensors and theirperception engine networks in accordance with one or moreimplementations;

FIG. 7 is a schematic diagram illustrating a system for training asecond beam steering radar from a first beam steering radar inaccordance with one or more implementations of the subject technology;

FIG. 8 is a flowchart for a GAN-based data synthesis for semi-supervisedlearning of a radar sensor in accordance with one or moreimplementations of the subject technology;

FIG. 9 is a flowchart for a method for semi-supervised training of aradar system in accordance with various implementations of the subjecttechnology; and

FIG. 10 conceptually illustrates an electronic system with which one ormore embodiments of the subject technology may be implemented.

DETAILED DESCRIPTION

Methods and apparatuses for implementing a Generative AdversarialNetwork (“GAN”) for synthesizing a training data set for a radar sensorare disclosed. The radar sensor is a beam steering radar capable ofgenerating narrow, directed beams that can be steered to any angle(i.e., from 0° to 360°) across a Field of View (“FoV”) to detectobjects. The beams are generated and steered in the analog domain, whileprocessing of received radar signals for object identification isperformed with advanced signal processing and machine learningtechniques. In various examples, the beam steering radar is used in anautonomous vehicle equipped with multiple sensors (e.g., camera, lidar,etc.) and a multi-sensor fusion platform for better control of drivingfunctions and a safer driving experience. Each sensor may be used todetect and identify objects with perception engines implemented withneural networks (e.g., deep learning networks). As described in moredetail herein below, a first beam steering radar is trained in asupervised manner with a set of training labels to generate a first setof radar object detection labels. This first set of radar objectdetection labels is then used in a GAN to generate a synthesizedtraining set to train a second beam steering radar with differentcharacteristics from the first beam steering radar.

It is appreciated that the detailed description set forth below isintended as a description of various configurations of the subjecttechnology and is not intended to represent the only configurations inwhich the subject technology may be practiced. The appended drawings areincorporated herein and constitute a part of the detailed description.The detailed description includes specific details for the purpose ofproviding a thorough understanding of the subject technology. However,the subject technology is not limited to the specific details set forthherein and may be practiced using one or more implementations. In one ormore instances, structures and components are shown in block diagramform in order to avoid obscuring the concepts of the subject technology.In other instances, well-known methods and structures may not bedescribed in detail to avoid unnecessarily obscuring the description ofthe examples. Also, the examples may be used in combination with eachother.

FIG. 1 illustrates an example environment in which a beam steering radarin an autonomous vehicle is used to detect and identify objects,according to various implementations of the subject technology. Egovehicle 100 is an autonomous vehicle with a beam steering radar system106 for transmitting a radar signal to scan a FoV or specific area. Asdescribed in more detail below, the radar signal is transmittedaccording to a set of scan parameters that can be adjusted to result inmultiple transmission beams 118. The scan parameters may include, amongothers, the total angle of the scanned area defining the FoV, the beamwidth or the scan angle of each incremental transmission beam, thenumber of chirps in the radar signal, the chirp time, the chirp segmenttime, the chirp slope, and so on. The entire FoV or a portion of it canbe scanned by a compilation of such transmission beams 118, which may bein successive adjacent scan positions or in a specific or random order.Note that the term FoV is used herein in reference to the radartransmissions and does not imply an optical FoV with unobstructed views.The scan parameters may also indicate the time interval between theseincremental transmission beams, as well as start and stop anglepositions for a full or partial scan.

In various examples, the ego vehicle 100 may also have other perceptionsensors, such as a camera 102 and a lidar 104. These perception sensorsare not required for the ego vehicle 100, but may be useful inaugmenting the object detection capabilities of the beam steering radar106. The camera 102 may be used to detect visible objects and conditionsand to assist in the performance of various functions. The lidar 104 canalso be used to detect objects and provide this information to adjustcontrol of the ego vehicle 100. This information may include informationsuch as congestion on a highway, road conditions, and other conditionsthat would impact the sensors, actions or operations of the vehicle.Existing ADAS modules utilize camera sensors to assist drivers indriving functions such as parking (e.g., in rear view cameras). Camerasare able to capture texture, color and contrast information at a highlevel of detail, but similar to the human eye, they are susceptible toadverse weather conditions and variations in lighting. The camera 102may have a high resolution but may not resolve objects beyond 50 meters.

Lidar sensors typically measure the distance to an object by calculatingthe time taken by a pulse of light to travel to an object and back tothe sensor. When positioned on top of a vehicle, a lidar sensor canprovide a 360° 3D view of the surrounding environment. Other approachesmay use several lidars at different locations around the vehicle toprovide the full 360° view. However, lidar sensors such as lidar 104 arestill prohibitively expensive, bulky in size, sensitive to weatherconditions and are limited to short ranges (e.g., less than 150-300meters). Radars, on the other hand, have been used in vehicles for manyyears and operate in all-weather conditions. Radar sensors also use farless processing than the other types of sensors and have the advantageof detecting objects behind obstacles and determining the speed ofmoving objects. When it comes to resolution, the laser beams emitted bythe lidar 104 are focused on small areas, have a smaller wavelength thanRF signals, and can achieve around 0.25 degrees of resolution.

In various examples and as described in more detail below, the beamsteering radar 106 can provide a 360° true 3D vision and human-likeinterpretation of the path and surrounding environment of the egovehicle 100. The beam steering radar 106 is capable of shaping andsteering RF beams in all directions in an FoV with at least one beamsteering antenna and recognize objects quickly and with a high degree ofaccuracy over a long range of around 300 meters or more. The short-rangecapabilities of the camera 102 and the lidar 104 along with thelong-range capabilities of the radar 106 enable a multi-sensor fusionmodule 108 in the ego vehicle 100 to enhance its object detection andidentification.

As illustrated, the beam steering radar 106 can detect both vehicle 120at a far range (e.g., greater than 350 m) as well as vehicles 110 and114 at a short range (e.g., lesser than 100 m). Detecting both vehiclesin a short amount of time and with enough range and velocity resolutionis imperative for full autonomy of driving functions of the ego vehicle.The radar 106 has an adjustable Long-Range Radar (“LRR”) mode thatenables the detection of long range objects in a very short time to thenfocus on obtaining finer velocity resolution for the detected vehicles.Although not described herein, radar 106 is capable oftime-alternatively reconfiguring between LRR and Short-Range Radar(“SRR”) modes. The SRR mode enables a wide beam with lower gain, and yetcan be configured to make quick decisions to avoid an accident, assistin parking and downtown travel, and capture information about a broadarea of the environment. The LRR mode enables narrow, directed beams toreach long distances and at a high gain; this is powerful for high speedapplications, and where longer processing time allows for greaterreliability. Excessive dwell time for each beam position may cause blindzones, and the adjustable LRR mode ensures that fast object detectioncan occur at long range while maintaining the antenna gain, transmitpower and desired Signal-to-Noise Ratio (SNR) for the radar operation.

Attention is now directed to FIG. 2, which illustrates an examplenetwork environment 200 in which a radar system may be implemented inaccordance with one or more implementations of the subject technology.The example network environment 200 includes a number of electronicdevices 220, 230, 240, 242, 244, 246, and 248 that are coupled to anelectronic device 210 via the transmission lines 250. The electronicdevice 210 may communicably couple the electronic devices 242, 244, 246,248 to one another. In one or more implementations, one or more of theelectronic devices 242, 244, 246, 248 are communicatively coupleddirectly to one another, such as without the support of the electronicdevice 210. Not all of the depicted components may be required, however,and one or more implementations may include additional components notshown in the figure. Variations in the arrangement and type of thecomponents may be made without departing from the scope of the claims asset forth herein. Additional components, different components, or fewercomponents may be provided.

In some implementations, one or more of the transmission lines 250 areEthernet transmission lines. In this respect, the electronic devices220, 230, 240, 242, 244, 246, 248 and 210 may implement a physical layer(PHY) that is interoperable with one or more aspects of one or morephysical layer specifications, such as those described in the Instituteof Electrical and Electronics Engineers (IEEE) 802.3 Standards (e.g.,802.3ch). The electronic device 210 may include a switch device, arouting device, a hub device, or generally any device that maycommunicably couple the electronic devices 220, 230, 240, 242, 244, 246,and 248.

In one or more implementations, at least a portion of the examplenetwork environment 200 is implemented within a vehicle, such as apassenger car. For example, the electronic devices 242, 244, 246, 248may include, or may be coupled to, various systems within a vehicle,such as a powertrain system, a chassis system, a telematics system, anentertainment system, a camera system, a sensor system, such as a lanedeparture system, a diagnostics system, or generally any system that maybe used in a vehicle. In FIG. 2, the electronic device 210 is depictedas a central processing unit, the electronic device 220 is depicted as aradar system, the electronic device 230 is depicted as a lidar systemhaving one or more lidar sensors, the electronic device 240 is depictedas an entertainment interface unit, and the electronic devices 242, 244,246, 248 are depicted as camera devices, such as forward-view, rear-viewand side-view cameras. In one or more implementations, the electronicdevice 210 and/or one or more of the electronic devices 242, 244, 246,248 may be communicatively coupled to a public communication network,such as the Internet.

The electronic device 210 includes a multi-sensor fusion platform forprocessing data acquired by electronic devices 220, 230, 240, 242, 244,246, and 248, including labeling objects detected and identified in theacquired data. Such objects may include structural elements in theenvironment near the vehicle such as roads, walls, buildings, roadcenter medians and other objects, as well as other vehicles,pedestrians, bystanders, cyclists, plants, trees, animals and so on.

FIG. 3 illustrates a schematic diagram of a beam steering radar systemimplemented as in FIG. 2 in accordance with various examples. Beamsteering radar 300 is a “digital eye” with true 3D vision and capable ofa human-like interpretation of the world. The “digital eye” andhuman-like interpretation capabilities are provided by two main modules:radar module 302 and a perception engine 304. Radar module 302 iscapable of both transmitting RF signals within a FoV and receiving thereflections of the transmitted signals as they reflect off of objects inthe FoV. With the use of analog beam steering in radar module 302, asingle transmit and receive chain can be used effectively to formdirectional, as well as steerable, beams.

The receive chain includes receive antennas 312 and 313, receive guardantennas 311 and 314, optional couplers 370-373, Low Noise Amplifiers(“LNAs”) 340-343, Phase Shifter (“PS”) circuits 320 and 322, amplifiers(such as Power Amplifiers (“PAs”)) 323, 324, 364 and 366, andcombination networks 344 and 345. The transmit chain includes drivers390, 392, 394 and 396, feed networks 334 and 336, PS circuits 316 and318, PAs 328-331, optional couplers 376, 378, 380 and 382, transmitantennas 308 and 309, and optional transmit guard antennas 307 and 310.The radar module 302 also includes a transceiver 306, aDigital-to-Analog (“DAC)” controller 390, a Field-Programmable GateArray (“FPGA”) 326, a microcontroller 338, processing engines 350, aGraphic User Interface (“GUI”) 658, temperature sensors 360 and adatabase 362. The processing engines 350 includes perception engine 304,database 352 and Digital Signal Processing (“DSP”) module 356. The DSPmodule 356 includes a monopulse module 357. Not all of the depictedcomponents may be required, however, and one or more implementations mayinclude additional components not shown in the figure. Variations in thearrangement and type of the components may be made without departingfrom the scope of the claims as set forth herein. Additional components,different components, or fewer components may be provided.

In operation, the transceiver 306 in radar module 302 generates signalsfor transmission through a series of transmit antennas 308 and 309 aswell as manages signals received through a series of receive antennas312 and 313. Beam steering within the FoV is implemented with PScircuits 316 and 318 coupled to the transmit antennas 308 and 309,respectively, on the transmit chain and PS circuits 320 and 322 coupledto the receive antennas 312 and 313, respectively, on the receive chain.Careful phase and amplitude calibration of the transmit antennas 308,309 and receive antennas 312, 313 can be performed in real-time with theuse of couplers integrated into the radar module 302 as described inmore detail below. In other implementations, calibration is performedbefore the radar is deployed in an ego vehicle and the couplers may beremoved.

The use of PS circuits 316, 318 and 320, 322 enables separate control ofthe phase of each element in the transmit antennas 308, 309 and receiveantennas 312, 313. Unlike early passive architectures, the beam issteerable not only to discrete angles but to any angle (i.e., from 0° to360°) within the FoV using active beamforming antennas. A multipleelement antenna can be used with an analog beamforming architecturewhere the individual antenna elements may be combined or divided at theport of the single transmit or receive chain without additional hardwarecomponents or individual digital processing for each antenna element.Further, the flexibility of multiple element antennas allows narrow beamwidth for transmit and receive. The antenna beam width decreases with anincrease in the number of antenna elements. A narrow beam improves thedirectivity of the antenna and provides the radar system 300 with asignificantly longer detection range.

A major challenge with implementing analog beam steering is to designPSs to operate at 77 GHz. PS circuits 316, 318 and 320, 322 solve thisproblem with a reflective PS design implemented with a distributedvaractor network fabricated using suitable semiconductor materials, suchas Gallium-Arsenide (GaAs) materials, among others. Each PS circuit 316,318 and 320, 322 has a series of PSs, with each PS coupled to an antennaelement to generate a phase shift value of anywhere from 0° to 360° forsignals transmitted or received by the antenna element. The PS design isscalable in future implementations to other semiconductor materials,such as Silicon-Germanium (SiGe) and CMOS, bringing down the PS cost tomeet specific demands of customer applications. Each PS circuit 316, 318and 320, 322 is controlled by an FPGA 326, which provides a series ofvoltages to the PSs in each PS circuit that results in a series of phaseshifts.

The DAC controller 390 is coupled to each of the LNAs 340-343, theamplifiers 323, 324, 364, 366, PS circuits 316, 318, 320, 322, thedrivers 390, 392, 394, 396, and the PAs 328-331. In someimplementations, the DAC controller 390 is coupled to the FPGA 326, andthe FPGA 326 can drive digital signaling to the DAC controller 390 toprovide analog signaling to the LNAs 340-343, the amplifiers 323, 324,364, 366, PS circuits 316, 318, 320, 322, the drivers 390, 392, 394,396, and the PAs 328-331. In some implementations, the DAC controller390 is coupled to the combination networks 344, 345 and to the feednetworks 334, 336.

In various examples, an analog control signal is applied to each PS inthe PS circuits 316, 318 and 320, 322 by the DAC controller 390 togenerate a given phase shift and provide beam steering. The analogcontrol signals applied to the PSs in PS circuits 316, 318 and 320, 322are based on voltage values that are stored in Look-up Tables (“LUTs”)in the FPGA 326. These LUTs are generated by an antenna calibrationprocess that determines which voltages to apply to each PS to generate agiven phase shift under each operating condition. Note that the PSs inPS circuits 316, 318 and 320, 322 can generate phase shifts at a veryhigh resolution of less than one degree. This enhanced control over thephase allows the transmit and receive antennas in radar module 302 tosteer beams with a very small step size, improving the capability of theradar system 300 to resolve closely located targets at small angularresolution. FPGA 326 also has LUTs to store bias voltage values for theLNAs 340-343. As described in more detail below, these bias voltagevalues can be determined during calibration to control the gain of theLNAs, including to vary the gain of LNAs connected to edge antennaelements of the receive antennas 312-313 in order to lower the side lobelevels of the received beams.

In various examples, each of the transmit antennas 308, 309 and thereceive antennas 312, 313 may be a meta-structure antenna, a phase arrayantenna, or any other antenna capable of radiating RF signals inmillimeter wave frequencies. A meta-structure, as generally definedherein, is an engineered structure capable of radiating electromagneticwaves in the mm-wave frequency range. Various configurations, shapes,designs and dimensions of the transmit antennas 308, 309 and the receiveantennas 312, 313 may be used to implement specific designs and meetspecific constraints.

The transmit chain in the radar module 302 starts with the transceiver306 generating RF signals to prepare for transmission over-the-air bythe transmit antennas 308 and 309. The RF signals may be, for example,Frequency-Modulated Continuous Wave (“FMCW”) signals. An FMCW signalenables the radar system 300 to determine both the range to an objectand the object's velocity by measuring the differences in phase orfrequency between the transmitted signals and the received/reflectedsignals or echoes. Within FMCW formats, there are a variety of waveformpatterns that may be used, including sinusoidal, triangular, sawtooth,rectangular and so forth, each having advantages and purposes.

Once the FMCW signals are generated by the transceiver 306, the FMCWsignals are fed to drivers 390 and 392. From the drivers 390 and 392,the signals are divided and distributed through feed networks 334 and336, respectively, which form a power divider system to divide an inputsignal into multiple signals, one for each element of the transmitantennas 308 and 309, respectively. The feed networks 334 and 336 maydivide the signals so power is equally distributed among them oralternatively, so power is distributed according to another scheme, inwhich the divided signals do not all receive the same power. Each signalfrom the feed networks 334 and 336 is then input to the PS circuits 316and 318, respectively, where the FMCW signals are phase shifted based oncontrol signaling from the DAC controller 390 (corresponding to voltagesgenerated by the FPGA 326 under the direction of microcontroller 338),and then transmitted to the PAs 329 and 330. Signal amplification isneeded for the FMCW signals to reach the long ranges desired for objectdetection, as the signals attenuate as they radiate by the transmitantennas 308 and 309. From the PAs 329 and 330, the FMCW signals are fedto couplers 378 and 380, respectively, to generate calibration signalingthat is fed back to the transceiver 306. From the couplers 378 and 380,the FMCW signals are transmitted through transmit antennas 308 and 309.Note that couplers 378-380 are used only for real-time calibrationpurposes and are therefore optional. Note also that, in someimplementations, the transceiver 306 feeds the FMCW signals to drivers394 and 396, which are then fed to PAs 328 and 332 and to the couplers376 and 382. From these couplers, the FMCW signals are fed to optionaltransmit guard antennas 307 and 310 for side lobe cancelation of thetransmission signal.

The microcontroller 338 determines which phase shifts to apply to thePSs in PS circuits 316, 318, 320 and 322 according to a desired scanningmode based on road and environmental scenarios. Microcontroller 338 alsodetermines the scan parameters for the transceiver to apply at its nextscan. The scan parameters may be determined at the direction of one ofthe processing engines 350, such as at the direction of perceptionengine 304. Depending on the objects detected, the perception engine 304may instruct the microcontroller 338 to adjust the scan parameters at anext scan to focus on a given area of the FoV or to steer the beams to adifferent direction.

In various examples and as described in more detail below, radar system300 operates in one of various modes, including a full scanning mode anda selective scanning mode, among others. In a full scanning mode, thetransmit antennas 308, 309 and the receive antennas 312, 313 can scan acomplete FoV with small incremental steps. Even though the FoV may belimited by system parameters due to increased side lobes as a functionof the steering angle, radar system 300 is able to detect objects over asignificant area for a long-range radar. The range of angles to bescanned on either side of boresight as well as the step size betweensteering angles/phase shifts can be dynamically varied based on thedriving environment. To improve performance of an autonomous vehicle(e.g., an ego vehicle) driving through an urban environment, the scanrange can be increased to keep monitoring the intersections and curbs todetect vehicles, pedestrians or bicyclists. This wide scan range maydeteriorate the frame rate (revisit rate) but is considered acceptableas the urban environment generally involves low velocity drivingscenarios. For a high-speed freeway scenario, where the frame rate iscritical, a higher frame rate can be maintained by reducing the scanrange. In this case, a few degrees of beam scanning on either side ofthe boresight would suffice for long-range target detection andtracking.

In a selective scanning mode, the radar system 300 scans around an areaof interest by steering to a desired angle and then scanning around thatangle. This ensures the radar system 300 is to detect objects in thearea of interest without wasting any processing or scanning cyclesilluminating areas with no valid objects. Since the radar system 300 candetect objects at a long distance, e.g., 300 m or more at boresight, ifthere is a curve in a road, direct measures do not provide helpfulinformation. Rather, the radar system 300 steers along the curvature ofthe road and aligns its beams towards the area of interest. In variousexamples, the selective scanning mode may be implemented by changing thechirp slope of the FMCW signals generated by the transceiver 306 and byshifting the phase of the transmitted signals to the steering anglesneeded to cover the curvature of the road.

Objects are detected with radar system 300 by reflections or echoes thatare received at the receive antennas 312 and 313. The received signalingis then optionally fed to couplers 372 and 373 using feedbackcalibration signaling from the transceiver 306. The couplers 370,372-374 can allow probing to the receive chain signal path duringreal-time calibration. From the couplers 372 and 373, the receivedsignaling is fed to LNAs 341 and 342. The LNAs 341 and 342 arepositioned between the receive antennas 312 and 313 and PS circuits 320and 322, which include PSs similar to the PSs in PS circuits 316 and318. For receive operation, PS circuits 320 and 322 create phasedifferentials between radiating elements in the receive antennas 312 and313 to compensate for the time delay of received signals betweenradiating elements due to spatial configurations.

Receive phase-shifting, also referred to as analog beamforming, combinesthe received signals for aligning echoes to identify the location, orposition of a detected object. That is, phase shifting aligns thereceived signals that arrive at different times at each of the radiatingelements in receive antennas 312 and 313. Similar to PS circuits 316,318 on the transmit chain, PS circuits 320, 322 are controlled by theDAC controller 390, which provides control signaling to each PS togenerate the desired phase shift. In some implementations, the FPGA 326can provide bias voltages to the DAC controller 390 to generate thecontrol signaling to PS circuits 320, 322.

The receive chain then combines the signals fed by the PS circuits 320and 322 at the combination networks 344 and 345, respectively, fromwhich the combined signals propagate to the amplifiers 364 and 366 forsignal amplification. The amplified signal is then fed to thetransceiver 306 for receiver processing. Note that as illustrated, thecombination networks 344 and 345 can generate multiple combined signals346 and 348, of which each signal combines signals from a number ofelements in the receive antennas 312 and 313, respectively. In oneexample, the receive antennas 312 and 313 include 128 and 64 radiatingelements partitioned into two 64-element and 32-element clusters,respectively. For example, the signaling fed from each cluster iscombined in a corresponding combination network (e.g., 344, 345) anddelivered to the transceiver 306 in a separate RF transmission line. Inthis respect, each of the combined signals 346 and 348 can carry two RFsignals to the transceiver 306, where each RF signal combines signalingfrom the 64-element and 32-element clusters of the receive antennas 312and 313. Other examples may include 8, 26, 34, or 62 elements, and soon, depending on the desired configuration. The higher the number ofantenna elements, the narrower the beam width. In some implementations,the receive guard antennas 311 and 314 feed the receiving signaling tocouplers 370 and 374, respectively, which are then fed to LNAs 340 and343. The filtered signals from the LNAs 340 and 343 are fed toamplifiers 323 and 324, respectively, which are then fed to thetransceiver 306 for side lobe cancelation of the received signals by thereceiver processing.

In some implementations, the radar module 302 includes receive guardantennas 311 and 314 that generate a radiation pattern separate from themain beams received by the 64-element receive antennas 312 and 313. Thereceive guard antennas 311 and 314 are implemented to effectivelyeliminate side-lobe returns from objects. The goal is for the receiveguard antennas 311 and 314 to provide a gain that is higher than theside lobes and therefore enable their elimination or reduce theirpresence significantly. The receive guard antennas 311 and 314effectively act as a side lobe filter. Similar, the radar module 302 mayoptionally include transmit guard antennas 307 and 310 to eliminate sidelobe formation or reduce the gain generated by transmitter side lobes atthe time of a transmitter main beam formation by the transmit antennas308 and 309.

Once the received signals are received by transceiver 306, the receivedsignals are processed by processing engines 350. Processing engines 350include perception engine 304 that detects and identifies objects in thereceived signal with one or more neural networks using machine learningor computer vision techniques, database 352 to store historical andother information for radar system 300, and the DSP module 354 with anADC module to convert the analog signals from transceiver 306 intodigital signals that can be processed by monopulse module 357 todetermine angles of arrival (AoA) and other valuable information for thedetection and identification of objects by perception engine 304. In oneor more implementations, DSP engine 356 may be integrated with themicrocontroller 338 or the transceiver 306.

Radar system 300 also includes a GUI 358 to enable configuration of scanparameters such as the total angle of the scanned area defining the FoV,the beam width or the scan angle of each incremental transmission beam,the number of chirps in the radar signal, the chirp time, the chirpslope, the chirp segment time, and so on as desired. In addition, radarsystem 300 has a temperature sensor 360 for sensing the temperaturearound the vehicle so that the proper voltages from FPGA 326 may be usedto generate the desired phase shifts. The voltages stored in FPGA 326are determined during calibration of the antennas under differentoperating conditions, including temperature conditions. A database 362may also be used in radar system 300 to store radar and other usefuldata.

The radar data may be organized in sets of Range-Doppler Map (RDM)information, corresponding to four-dimensional (4D) information that isdetermined by each RF beam reflected from targets, such as azimuthalangles, elevation angles, range, and velocity. The RDMs may be extractedfrom FMCW radar signals and may contain both noise and systematicartifacts from Fourier analysis of the radar signals. The perceptionengine 304 controls further operation of the transmit antennas 308 and309 by, for example, providing an antenna control signal containing beamparameters for the next RF beams to be radiated from the transmitantennas 308-309.

In operation, the microcontroller 338 may, for example, determine theparameters at the direction of perception engine 304, which may at anygiven time determine to focus on a specific area of an FoV uponidentifying targets of interest in the ego vehicle's path or surroundingenvironment. The microcontroller 338 determines the direction, power,and other parameters of the RF beams and controls the transmit antennas308 and 309 to achieve beam steering in various directions. Next, thetransmit antennas 308 and 309 radiate RF beams having the determinedparameters. The RF beams are reflected from targets in and around theego vehicle's path (e.g., in a 360° field of view) and are received bythe transceiver 306. The receive antennas 312 and 313 send the receivedRF beams to the transceiver 306 for generating the 4D radar data for theperception engine 304 for target identification.

In various examples, the perception engine 304 can store informationthat describes an FoV. This information may be historical data used totrack trends and anticipate behaviors and traffic conditions or may beinstantaneous or real-time data that describes the FoV at a moment intime or over a window in time. The ability to store this data enablesthe perception engine 304 to make decisions that are strategicallytargeted at a particular point or area within the FoV. For example, theFoV may be clear (e.g., no echoes received) for a period of time (e.g.,five minutes), and then one echo arrives from a specific region in theFoV; this is similar to detecting the front of a car. In response, theperception engine 304 may determine to narrow the beam width for a morefocused view of that sector or area in the FoV. The next scan mayindicate the targets' length or other dimension, and if the target is avehicle, the perception engine 304 may consider what direction thetarget is moving and focus the beams on that area. Similarly, the echomay be from a spurious target, such as a bird, which is small and movingquickly out of the path of the vehicle. The database 352 coupled to theperception engine 304 can store useful data for radar system 300, suchas, for example, information on which subarrays of the transmit antennas308 and 309 perform better under different conditions.

In various examples described herein, the use of radar system 300 in anautonomous driving vehicle provides a reliable way to detect targets indifficult weather conditions. For example, historically a driver willslow down dramatically in thick fog, as the driving speed decreasesalong with decreases in visibility. On a highway in Europe, for example,where the speed limit is 515 km/h, a driver may need to slow down to 50km/h when visibility is poor. Using the radar system 300, the driver (ordriverless vehicle) may maintain the maximum safe speed without regardto the weather conditions. Even if other drivers slow down, a vehicleenabled with the radar system 300 can detect those slow-moving vehiclesand obstacles in its path and avoid/navigate around them.

Additionally, in highly congested areas, it is necessary for anautonomous vehicle to detect targets in sufficient time to react andtake action. The examples provided herein for a radar system increasethe sweep time of a radar signal to detect any echoes in time to react.In rural areas and other areas with few obstacles during travel, theperception engine 304 adjusts the focus of the RF beam to a larger beamwidth, thereby enabling a faster scan of areas where there are fewechoes. The perception engine 304 may detect this situation byevaluating the number of echoes received within a given time period andmaking beam size adjustments accordingly. Once a target is detected, theperception engine 304 determines how to adjust the beam focus.

In various examples, beam steering radar 300 is used in an autonomousdriving system that provides some or full automation of drivingfunctions for an ego vehicle (e.g., ego vehicle 100 of FIG. 1) asillustrated in FIG. 4. The driving functions may include, for example,steering, accelerating, braking, and monitoring the surroundingenvironment and driving conditions to respond to events, such aschanging lanes or speed when needed to avoid traffic, crossingpedestrians, animals, and so on. As shown in FIG. 4, the autonomousdriving system 400 includes multiple sensors, such as beam steeringradars 402-404, cameras 406-408, lidars 410-412, infrastructure sensors414, environmental sensors 416, operational sensors 418, user preferencesensors 420, and other sensors 422. The autonomous driving system 400also includes a communications module 430, a multi-sensor fusionplatform 432, a system controller 424, a system memory 426, and aVehicle-to-Vehicle (V2V) communications module 426. It is appreciatedthat this configuration of autonomous driving system 400 is an exampleconfiguration and not meant to be limiting to the specific structureillustrated in FIG. 4. Additional systems and modules not shown in FIG.4 may be included in autonomous driving system 400.

As illustrated, the autonomous driving system 400 has multiple camerasensors 406-408 positioned in different locations in the ego vehicle,such as cameras 242-248 of FIG. 2. The autonomous driving system 400also has at least one lidar sensor to create a 360° 3D map of theenvironment represented as a point cloud. In various implementations,the system 400 has a lidar 408 that measures distances to objectsdetected and identified in the 3D map and a more sophisticated lidar 410that is also able to measure the velocity of moving objects, such as aFrequency Modulated Continuous Wave (“FMCW”) lidar.

In various examples, the beam steering radars 402-404 are implemented asbeam steering radar 300 and include at least one beam steering antennafor providing dynamically controllable and steerable beams that canfocus on one or multiple portions of a 360° F.oV of the vehicle. Thebeams radiated from the beam steering antenna are reflected back fromobjects in the vehicle's path and surrounding environment and receivedand processed by a perception engine coupled to the beam steering radars402-404 to detect and identify the objects and control the radar moduleas desired. The beam steering radars 402-404 may have differenttransceiver and DSP chips with different properties or different scanpatterns. One or both radars may be in operation at any given time. Inaddition, both the camera sensors 406-408 and lidars 410-412 are alsocoupled to perception engines having neural networks capable ofdetecting and identifying objects in the acquired data.

Infrastructure sensors 414 may provide information from infrastructurewhile driving, such as from a smart road configuration, bill boardinformation, traffic alerts and indicators, including traffic lights,stop signs, traffic warnings, and so forth. Environmental sensors 416detect various conditions outside, such as temperature, humidity, fog,visibility, precipitation, among others. Operational sensors 418 provideinformation about the functional operation of the vehicle. This may betire pressure, fuel levels, brake wear, and so on. The user preferencesensors 420 may detect conditions that are part of a user preference.This may be temperature adjustments, smart window shading, etc. Othersensors 422 may include additional sensors for monitoring conditions inand around the ego vehicle.

In various examples, the multi-sensor fusion platform 432 optimizesthese various functions to provide an approximately comprehensive viewof the ego vehicle and environments. Many types of sensors may becontrolled by the multi-sensor fusion platform 432. These sensors maycoordinate with each other to share information and consider the impactof one control action on another system. In one example, in a congesteddriving condition, a noise detection module (not shown) may identifythat there are multiple radar signals that may interfere with thevehicle. This information may be used by the perception engine coupledto the beam steering radars 402-404 to adjust the scan parameters of thebeam steering radars 402-404 to avoid these other signals and minimizeinterference.

In another example, environmental sensor 416 may detect that the weatheris changing, and visibility is decreasing. In this situation, themulti-sensor fusion platform 432 may determine to configure the othersensors to improve the ability of the vehicle to navigate in these newconditions. The configuration may include turning off the cameras402-404 and/or the lidars 410-412 or reducing the sampling rate of thesevisibility-based sensors. This effectively places reliance on thesensor(s) adapted for the current situation. In response, the perceptionengines configure the sensors for these conditions as well. For example,the beam steering radars 402-404 may reduce the beam width to provide amore focused beam, and thus a finer sensing capability.

In various examples, the multi-sensor fusion platform 432 may send adirect control to the radars 402-404 based on historical conditions andcontrols. The multi-sensor fusion platform 432 may also use some of thesensors within the autonomous driving system 400 to act as feedback orcalibration for the other sensors. In this way, the operational sensors418 may provide feedback to the multi-sensor fusion platform 432 tocreate templates, patterns and control scenarios. These are based onsuccessful actions or may be based on poor results, where themulti-sensor fusion platform 432 learns from past actions.

Data from the sensors 402-422 may be combined in the multi-sensor fusionplatform 432 to improve the target detection and identificationperformance of autonomous driving system 400. The multi-sensor fusionplatform 432 may itself be controlled by the system controller 424,which may also interact with and control other modules and systems inthe ego vehicle. For example, the system controller 424 may power on oroff the different sensors 402-422 as desired, or provide instructions tothe ego vehicle to stop upon identifying a driving hazard (e.g., deer,pedestrian, cyclist, or another vehicle suddenly appearing in thevehicle's path, flying debris, etc.)

All modules and systems in the autonomous driving system 400 communicatewith each other through the communication module 430. The system memory426 may store information and data (e.g., static and dynamic data) usedfor operation of the autonomous driving system 400 and the ego vehicleusing the autonomous driving system 400. The V2V communications module430 is used for communication with other vehicles, such as to obtaininformation from other vehicles that is non-transparent to the user,driver, or rider of the ego vehicle, and to help vehicles coordinatewith one another to avoid any type of collision.

A challenge in the detection and identification of objects withautonomous driving system 400 is in the training of the neural networkscoupled to the beam steering radar sensors 402-404. Training of neuralnetworks for object recognition tasks often requires considerable dataand reliable training sets having labeled data. As the beam steeringradar data is in the form of range-doppler maps (“RDMs”) with range andvelocity information but no apparent features or characteristics ofobjects, labelling of the data is not an intuitive task.

FIG. 5 illustrates a range-doppler map captured by a beam steering radarsuch as beam steering radars 402-404. Beam steering radars 402-404 areable to capture data relating to the cars, pedestrians and cyclists inthe scene 502, whether in movement or stationary. For ease ofunderstanding, the objects detected in scene 502 are shown in therange-doppler map with bounded boxes, e.g., bounded box 504. Initially,however, all data acquired by the radars 402-404 is in the form ofunlabeled data. The RDMs display only the range and velocity informationof detected objects on a x-y axis, with objects appearing as a clusterof similar intensity pixels. As such, these unlabeled RDMs do notindicate the category of the objects detected. There is no intuitive wayof knowing which objects appear on an unlabeled RDM or even which pixelsin the RDM correspond to detected objects.

In order to properly identify objects in RDMs generated by a beamsteering radar described herein, the perception engine coupled to theradar may be initially trained with labeled data from lidar and camerasensors. FIG. 6 illustrates various sensors and their perception enginenetworks in accordance with one or more implementations. Data fromcamera 600 is fed into a camera network 602 for generating camera objectdetection labels 604. Similarly, data from lidar 606 is fed into a lidarnetwork 608 for generating lidar object detection labels 610. Thelabeled data from camera 600 and lidar 606 is created from data acquiredduring actual driving conditions on various roads, time of day andweather patterns. Initially, human observers may label the data to traincamera network 602 and lidar network 608 to identify objects in thedata. Once the networks 602 and 608 are trained, the generated cameraobject detection labels 604 and lidar object detection labels 610 may beused in the training of radar network 614 for labeling data from beamsteering radar 612.

As described herein below, there may be multiple beam steering radarswith different characteristics, configurations and scan patternsdeployed in a vehicle. Once a preliminary training of the first beamsteering radar 612 is performed, training of the first beam steeringradar 612 may occur during the training of camera 600 and lidar 604 onactual driving conditions. In some embodiments, training of the firstbeam steering radar 612 may occur in parallel with the training ofcamera 600 and lidar 604 on actual driving conditions. The first set ofradar object detection labels 616 may be used to train a GAN to generatea synthesized training set to train a second beam steering radar network620 for a second beam steering radar 618 with differing characteristicsfrom the first beam steering radar 612. Note that this facilitatesdeployment of the second beam steering radar 618 as its training can beperformed in advance of deployment by leveraging the training of thefirst beam steering radar 612.

FIG. 7 illustrates a system for training a second beam steering radarfrom a first beam steering radar in more detail. In various examples,the training of a second beam steering radar from a first beam steeringradar is accomplished by implementing a GAN module 700. A GAN is acombination of two recurrent neural networks that can generate a new,synthetic instance of data that can pass for real data: a generativenetwork for the raw radar data and a discriminative network forestimating the probability that a labeled data set is in fact real data.The generative network generates new data instances while thediscriminative network decides whether the labels that it reviews belongto the actual training data set. Over time, the two networks become moreaccurate in producing a synthetic training data set that appearsauthentic.

Training of the GAN module 700 proceeds in alternating periods, with thediscriminative network being trained in one period to learn how todistinguish real data from fake data and the generative network beingtrained in a second period to learn how to generate better instances ofdata. The discriminative network takes in both real data and fake dataand outputs probabilities with a number between 0 and 1, with 1representing a prediction of authenticity as real data and 0representing fake data. In accordance with some embodiments, thediscriminative network is a binary classifier that outputs real data as1 and fake data as 0. As for example, real data is data that acquiredfrom a beam steering radar whereas fake data can be any generated set ofdata, which can be random or simulated real-time acquired data, that isfed into the discriminative network to train the GAN module. The realdata fed into the GAN module 700 comes from the first beam steeringradar that has already been trained, i.e., data 702 and itscorresponding object detection labels 704. The GAN module 700 must learnhow to distinguish this real data from fake data, which initially can bea random data set or a small data set 704 from the second beam steeringradar. Once the GAN module 700 is trained, it is ready to generate asynthetic data set for the second beam steering radar. The syntheticdata set is composed of data set 712 and corresponding labels 714,generated during inference from actual data from the first beam steeringradar, that is, generated from data set 708 and its corresponding labels710. The synthetic data set 712-714 is used to train radar network 716for the second beam steering radar, which can then be deployed in an egovehicle to detect and identify objects on the fly. Note thatimplementing the GAN module 700 enables the use of two beam steeringradars with different characteristics while saving time andcomputational resources to train them. The first beam steering radar istrained to then facilitate and speed up the training of the second beamsteering radar.

A flowchart illustrating this process is shown in FIG. 8, in accordancewith various implementations of the subject technology. As illustratedin FIG. 8, a first radar network is trained to generate a first set ofradar object detection labels corresponding to a first radar data set(802). Once the first beam steering radar is trained, a GAN is trainedto synthesize a training set for a second beam steering radar from thefirst set of radar object detection labels and the first radar data set(804). The synthesized training set is then used to generate a secondset of radar object detection labels identifying objects correspondingto a second radar set detected by the second beam steering radar (806).

FIG. 9 is a flowchart for a method 900 for semi-supervised training of aradar system in accordance with various implementations of the subjecttechnology. For explanatory purposes, the method 900 of FIG. 9 isdescribed herein with reference to various radar systems and platformsdescribed with respect to FIGS. 1-7; however, the method 900 is notlimited to the such platforms and the radar systems.

As illustrated in FIG. 9, the method 900 includes training a first radarnetwork of the radar system, at step 910. The first radar network can beconfigured to provide a first set of radar object detection labelscorresponding to a first set of radar data. In accordance with variousembodiments herein, the first set of radar data can be acquired via oneor more of lidar or camera sensors of the radar system. This can be doneduring an actual driving condition that occurs on a plurality of roads,at various times of the day for one or more days, or under variousweather patterns. The one or more of lidar and/or camera sensors and thebeam steering radar are components in an autonomous vehicle. Theautonomous vehicle may further include additional components or modulesas described herein. In some embodiments, the radar system includes oneor more beam steering radars as disclosed herein.

At step 920, the method 900 includes training a generative adversarialnetwork (GAN) with the trained first radar network. As described herein,the GAN or GAN module is a neural network that can be configured todistinguish real data from fake data as described above with respect toFIGS. 6 and 7. In various implementations, the real data can comprisethe first set of radar data obtained during actual driving conditionsthat is acquired via one or more of lidar or camera sensors of the radarsystem. In various embodiments, the GAN or GAN module can include acombination of a generative network and a discriminative network. Inaccordance with various embodiments, the generative network can beconfigured for generating new data instances based on the first set ofradar data and the discriminative network can be configured todistinguish the new data instances from the fake data.

The method 900 includes, at step 930, synthesizing a training data setfor a second radar network of the radar system with the trained GAN. Invarious implementations, the synthesized training data set can comprisea data set and corresponding labels of the data set that are generatedduring inference from the first set of radar and the first set of radarobject detection labels, as described above with respect to FIGS. 6 and7.

At step 940, the method 900 includes training a second radar networkwith the synthesized training data set. In various embodiments, thefirst radar network includes a first beam steering radar and the secondradar network includes a second beam steering radar. In variousimplementations, the first beam steering radar and the second beamsteering radar can have different characteristics in at least one of aconfiguration of parameters or a scan pattern. In variousimplementations, the training of the second radar network by the GAN isperformed prior to deployment of the second beam steering radar on anautonomous vehicle.

At step 950, the method 900 includes generating a second set of radarobject detection labels based on the training of the second radarnetwork. In addition, the method 900 may optionally include training oneor more of lidar or camera sensors on actual driving conditions, whereinthe training of the one or more of lidar or camera sensors occurs duringthe training of the first radar network of the radar system. Inaccordance with various implementations and embodiments herein, any ofthe methods or processes described with respect to FIGS. 6-8 can beincluded in the method 900 as described with respect to FIG. 9.

FIG. 10 conceptually illustrates an electronic system 1000 with whichone or more embodiments of the subject technology may be implemented.For example, each of the perception engines, including but not limitedto, for example, a radar perception engine, a camera perception engine,and a lidar perception engine, along with their respective neuralnetworks, including but not limited to, for example, a radar neuralnetwork, a camera neural network, and a lidar neural network, may beimplemented via the electronic system 1000 or via any of the componentswithin the electronic system 1000. In addition, the electronic system1000 or any of the components within the electronic system 1000 can beconfigured to process any of methods 800 or 900 to perform the variousmethod or process steps in their respective methods, including, forexample, but not limited to method or process steps related to training,retraining, generating, regenerating, accessing, and acquiring.

As illustrated in FIG. 10, the electronic system 1000, for example, canbe a computer, a server, or generally any electronic device thatexecutes a program. Such an electronic system includes various types ofcomputer readable media and interfaces for various other types ofcomputer readable media. The electronic system 1000 includes a bus 1008,one or more processing unit(s) 1012, a system memory 1004 (and/orbuffer), a read-only memory (ROM) 1010, a permanent storage device 1002,an input device interface 1014, an output device interface 1006, and oneor more network interfaces 1016, or subsets and variations thereof.

The bus 1008 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 1000. In one or more implementations, the bus 1008communicatively connects the one or more processing unit(s) 1012 withthe ROM 1010, the system memory 1004, and the permanent storage device1002. From these various memory units, the one or more processingunit(s) 1012 retrieves instructions to execute and data to process inorder to execute the processes of the subject disclosure. For example,the processing unit(s) 1012 can execute instructions that perform one ormore processes, such as from methods 800 and 900. The one or moreprocessing unit(s) 1012 can be a single processor or a multi-coreprocessor in different implementations.

The ROM 1010 stores static data and instructions that are needed by theone or more processing unit(s) 1012 and other modules of the electronicsystem 1000. The permanent storage device 1002, on the other hand, maybe a read-and-write memory device. The permanent storage device 1002 maybe a non-volatile memory unit that stores instructions and data evenwhen the electronic system 1000 is off. In one or more implementations,a mass-storage device (such as a magnetic or optical disk and itscorresponding disk drive) may be used as the permanent storage device1002.

In one or more implementations, a removable storage device (such as afloppy disk, flash drive, and its corresponding disk drive) may be usedas the permanent storage device 1002. Like the permanent storage device1002, the system memory 1004 may be a read-and-write memory device.However, unlike the permanent storage device 1002, the system memory1004 may be a volatile read-and-write memory, such as a random accessmemory. The system memory 1004 may store any of the instructions anddata that one or more processing unit(s) 1012 may need at runtime. Inone or more implementations, the processes of the subject disclosure arestored in the system memory 1004, the permanent storage device 1002,and/or the ROM 1010. From these various memory units, the one or moreprocessing unit(s) 1012 retrieves instructions to execute and data toprocess in order to execute the processes of one or moreimplementations.

The bus 1008 also connects to the input and output device interfaces1014 and 1006. The input device interface 1014 enables a user tocommunicate information and select commands to the electronic system1000. Input devices that may be used with the input device interface1014 may include, for example, alphanumeric keyboards and pointingdevices (also called “cursor control devices”). The output deviceinterface 1006 may enable, for example, the display of images generatedby electronic system 1000. Output devices that may be used with theoutput device interface 1006 may include, for example, printers anddisplay devices, such as a liquid crystal display (LCD), a lightemitting diode (LED) display, an organic light emitting diode (OLED)display, a flexible display, a flat panel display, a solid-statedisplay, a projector, or any other device for outputting information.One or more implementations may include devices that function as bothinput and output devices, such as a touchscreen. In theseimplementations, feedback provided to the user can be any form ofsensory feedback, such as visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input.

Finally, as shown in FIG. 10, the bus 1008 also couples the electronicsystem 1000 to a network (not shown) and/or to one or more devicesthrough the one or more network interface(s) 1016, such as one or morewireless network interfaces. In this manner, the electronic system 1000can be a part of a network of computers (such as a local area network(“LAN”), a wide area network (“WAN”), or an Intranet, or a network ofnetworks, such as the Internet. Any or all components of the electronicsystem 1000 can be used in conjunction with the subject disclosure.

In accordance with various embodiments, a method for semi-supervisedtraining of a radar system is described. The method for semi-supervisedtraining of a radar system includes training a first radar network ofthe radar system with a first set of radar object detection labelscorresponding to a first set of radar data; training a generativeadversarial network (GAN) with the trained first radar network;synthesizing a training data set for a second radar network of the radarsystem with the trained GAN; training a second radar network with thesynthesized training data set; and generating a second set of radarobject detection labels based on the training of the second radarnetwork.

In various embodiments, the first set of radar data is acquired via oneor more of lidar or camera sensors of the radar system during an actualdriving condition that occurs on a plurality of roads at various timeswith various weather patterns for one or more days. In variousembodiments, the method may optionally include training one or more oflidar or camera sensors on actual driving conditions, wherein thetraining of the one or more of lidar or camera sensors occurs during thetraining of the first radar network of the radar system.

In various embodiments, the synthesized training data set includes adata set and corresponding labels that are generated during an inferencefrom the first set of radar and the first set of radar object detectionlabels. In various embodiments, the GAN is a neural network configuredto distinguish real data from fake data, wherein the real data comprisesthe first set of radar data obtained during actual driving conditionsacquired via one or more of lidar or camera sensors of the radar system.In various embodiments, the GAN includes a combination of a generativenetwork and a discriminative network. In some embodiments, thegenerative network is configured for generating new data instances basedon the first set of radar data and the discriminative network isconfigured to distinguish the new data instances from the fake data.

In various embodiments, the first radar network includes a first beamsteering radar and the second radar network includes a second beamsteering radar. In various embodiments, the first beam steering radarand the second beam steering radar have different characteristics in atleast one of a configuration of parameters or a scan pattern. In variousembodiments, the training of the second radar network by the GAN isperformed prior to deployment of the second beam steering radar.

In accordance with various embodiments, a system for training a radar ora radar system is disclosed. The system includes a first radar networkthat provides a first set of radar object detection labels correspondingto a first set of radar data; a second radar network; and a GAN moduleconfigured to train the second radar network using the first set ofradar object detection labels and the first set of radar data.

In various embodiments, the first set of radar data is acquired via oneor more of lidar or camera sensors of the radar during an actual drivingcondition that occurs on a plurality of roads at various times withvarious weather patterns for one or more days. In various embodiments,the GAN module can include a neural network configured to distinguishreal data from fake data, wherein the real data comprises the first setof radar data obtained during actual driving conditions acquired via oneor more of lidar or camera sensors of the radar. In various embodiments,the GAN module can include a combination of a generative network and adiscriminative network, wherein the generative network is configured forgenerating new data instances based on the first set of radar data andthe discriminative network is configured to distinguish the new datainstances from the fake data. In various embodiments, the GAN module isconfigured to synthesize a training set for the second radar network ofthe radar using the first set of radar object detection labels and thefirst set of radar data.

In various embodiments, the first radar network comprises a first beamsteering radar and the second radar network comprises a second beamsteering radar, wherein the first beam steering radar and the secondbeam steering radar have different characteristics in at least one of aconfiguration of parameters or a scan pattern. In various embodiments,the second radar network is trained with the GAN module prior todeployment of the second beam steering radar.

In accordance with various embodiments, a non-transitory computerreadable medium is disclosed. The medium includes computer executableinstructions stored thereon to cause one or more processing units totrain a first radar network of a radar system with a first set of radarobject detection labels corresponding to a first set of radar data;train a generative adversarial network (GAN) with the trained firstradar network; synthesize a training data set for a second radar networkof the radar system with the trained GAN; train a second radar networkwith the synthesized training data set; and generate a second set ofradar object detection labels based on the training of the second radarnetwork.

In various embodiments, the first set of radar data is acquired via oneor more of lidar or camera sensors of the radar system during an actualdriving condition that occurs on a plurality of roads at various timeswith various weather patterns for one or more days. In variousembodiments, the medium includes further instructions to cause one ormore processing units to train one or more of lidar or camera sensors onactual driving conditions, wherein the training of the one or more oflidar or camera sensors occurs during the training of the first radarnetwork of the radar system.

In various embodiments, the GAN is a neural network configured todistinguish real data from fake data, wherein the GAN comprises acombination of a generative network and a discriminative network,wherein the generative network is configured for generating new datainstances based on the first set of radar data and the discriminativenetwork is configured to distinguish the new data instances from thefake data.

In various embodiments, the first radar network comprises a first beamsteering radar and the second radar network comprises a second beamsteering radar, wherein the first beam steering radar and the secondbeam steering radar have different characteristics in at least one of aconfiguration of parameters or a scan pattern, and wherein the trainingof the second radar network by the GAN is performed prior to deploymentof the second beam steering radar.

It is appreciated that the beam steering radar described herein abovesupports autonomous driving with improved sensor performance,all-weather/all-condition detection, advanced decision-making algorithmsand interaction with other sensors through sensor fusion. Theseconfigurations optimize the use of radar sensors, as radar is notinhibited by weather conditions in many applications, such as forself-driving cars. The radar described here is effectively a “digitaleye,” having true 3D vision and capable of human-like interpretation ofthe world.

The previous description of the disclosed examples is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these examples will be readily apparent tothose skilled in the art, and the m spirit or scope of the disclosure.Thus, the present disclosure is not intended to be limited to theexamples shown herein but is to be accorded the widest scope consistentwith the principles and novel features disclosed herein.

As used herein, the phrase “at least one of” preceding a series ofitems, with the terms “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (i.e.,each item). The phrase “at least one of” does not require selection ofat least one item; rather, the phrase allows a meaning that includes atleast one of any one of the items, and/or at least one of anycombination of the items, and/or at least one of each of the items. Byway of example, the phrases “at least one of A, B, and C” or “at leastone of A, B, or C” each refer to only A, only B, or only C; anycombination of A, B, and C; and/or at least one of each of A, B, and C.

Furthermore, to the extent that the term “include,” “have,” or the likeis used in the description or the claims, such term is intended to beinclusive in a manner similar to the term “comprise” as “comprise” isinterpreted when employed as a transitional word in a claim.

A reference to an element in the singular is not intended to mean “oneand only one” unless specifically stated, but rather “one or more.” Theterm “some” refers to one or more. Underlined and/or italicized headingsand subheadings are used for convenience only, do not limit the subjecttechnology, and are not referred to in connection with theinterpretation of the description of the subject technology. Allstructural and functional equivalents to the elements of the variousconfigurations described throughout this disclosure that are known orlater come to be known to those of ordinary skill in the art areexpressly incorporated herein by reference and intended to beencompassed by the subject technology. Moreover, nothing disclosedherein is intended to be dedicated to the public regardless of whethersuch disclosure is explicitly recited in the above description.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of what may be claimed, but ratheras descriptions of particular implementations of the subject matter.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable sub combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to a subcombination or variation of a sub combination.

The subject matter of this specification has been described in terms ofparticular aspects, but other aspects can be implemented and are withinthe scope of the following claims. For example, while operations aredepicted in the drawings in a particular order, this should not beunderstood as requiring that such operations be performed in theparticular order shown or in sequential order, or that all illustratedoperations be performed, to achieve desirable results. The actionsrecited in the claims can be performed in a different order and stillachieve desirable results. As one example, the processes depicted in theaccompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. Moreover, theseparation of various system components in the aspects described aboveshould not be understood as requiring such separation in all aspects,and it should be understood that the described program components andsystems can generally be integrated together in a single hardwareproduct or packaged into multiple hardware products. Other variationsare within the scope of the following claim.

What is claimed is:
 1. A method for semi-supervised training of a radarsystem, comprising: training a first radar network of the radar systemwith a first set of radar object detection labels corresponding to afirst set of radar data; training a generative adversarial network (GAN)with the trained first radar network; synthesizing a training data setfor a second radar network of the radar system with the trained GAN;training a second radar network with the synthesized training data set;and generating a second set of radar object detection labels based onthe training of the second radar network.
 2. The method of claim 1,wherein the first set of radar data is acquired via one or more of lidaror camera sensors of the radar system during an actual driving conditionthat occurs on a plurality of roads at various times with variousweather patterns for one or more days.
 3. The method of claim 1, furthercomprising: training one or more of lidar or camera sensors on actualdriving conditions, wherein the training of the one or more of lidar orcamera sensors occurs during the training of the first radar network ofthe radar system.
 4. The method of claim 1, wherein the synthesizedtraining data set comprises a data set and corresponding labels that aregenerated during an inference from the first set of radar and the firstset of radar object detection labels.
 5. The method of claim 1, whereinthe GAN is a neural network configured to distinguish real data fromfake data, wherein the real data comprises the first set of radar dataobtained during actual driving conditions acquired via one or more oflidar or camera sensors of the radar system.
 6. The method of claim 5,wherein the GAN comprises a combination of a generative network and adiscriminative network, wherein the generative network is configured forgenerating new data instances based on the first set of radar data andthe discriminative network is configured to distinguish the new datainstances from the fake data.
 7. The method of claim 1, wherein thefirst radar network comprises a first beam steering radar and the secondradar network comprises a second beam steering radar, wherein the firstbeam steering radar and the second beam steering radar have differentcharacteristics in at least one of a configuration of parameters or ascan pattern.
 8. The method of claim 7, wherein the training of thesecond radar network by the GAN is performed prior to a deployment ofthe second beam steering radar.
 9. A system for training a radar,comprising: a first radar network that provides a first set of radarobject detection labels corresponding to a first set of radar data; asecond radar network; and a GAN module configured to train the secondradar network using the first set of radar object detection labels andthe first set of radar data.
 10. The system of claim 9, wherein thefirst set of radar data is acquired via one or more of lidar or camerasensors of the radar during an actual driving condition that occurs on aplurality of roads at various times with various weather patterns forone or more days.
 11. The system of claim 9, wherein the GAN modulecomprises a neural network configured to distinguish real data from fakedata, wherein the real data comprises the first set of radar dataobtained during actual driving conditions acquired via one or more oflidar or camera sensors of the radar.
 12. The system of claim 11,wherein the GAN module comprises a combination of a generative networkand a discriminative network, wherein the generative network isconfigured for generating new data instances based on the first set ofradar data and the discriminative network is configured to distinguishthe new data instances from the fake data.
 13. The system of claim 9,wherein the GAN module is configured to synthesize a training set forthe second radar network of the radar using the first set of radarobject detection labels and the first set of radar data.
 14. The systemof claim 9, wherein the first radar network comprises a first beamsteering radar and the second radar network comprises a second beamsteering radar, wherein the first beam steering radar and the secondbeam steering radar have different characteristics in at least one of aconfiguration of parameters or a scan pattern.
 15. The system of claim14, wherein the second radar network is trained with the GAN moduleprior to a deployment of the second beam steering radar.
 16. Anon-transitory computer readable medium comprising computer executableinstructions stored thereon to cause one or more processing units to:train a first radar network of a radar system with a first set of radarobject detection labels corresponding to a first set of radar data;train a generative adversarial network (GAN) with the trained firstradar network; synthesize a training data set for a second radar networkof the radar system with the trained GAN; train a second radar networkwith the synthesized training data set; and generate a second set ofradar object detection labels based on the training of the second radarnetwork.
 17. The non-transitory computer readable medium of claim 16,wherein the first set of radar data is acquired via one or more of lidaror camera sensors of the radar system during an actual driving conditionthat occurs on a plurality of roads at various times with variousweather patterns for one or more days.
 18. The non-transitory computerreadable medium of claim 16, wherein the instructions further cause oneor more processing units to: train one or more of lidar or camerasensors on actual driving conditions, wherein the training of the one ormore of lidar or camera sensors occurs during the training of the firstradar network of the radar system.
 19. The non-transitory computerreadable medium of claim 16, wherein the GAN is a neural networkconfigured to distinguish real data from fake data, wherein the GANcomprises a combination of a generative network and a discriminativenetwork, wherein the generative network is configured for generating newdata instances based on the first set of radar data and thediscriminative network is configured to distinguish the new datainstances from the fake data.
 20. The non-transitory computer readablemedium of claim 16, wherein the first radar network comprises a firstbeam steering radar and the second radar network comprises a second beamsteering radar, wherein the first beam steering radar and the secondbeam steering radar have different characteristics in at least one of aconfiguration of parameters or a scan pattern, and wherein the trainingof the second radar network by the GAN is performed prior to deploymentof the second beam steering radar.